ability to async read chunk data , fix leak data error for react native #5522
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Diff output filesdiff --git a/packages/@uppy/aws-s3/lib/HTTPCommunicationQueue.js b/packages/@uppy/aws-s3/lib/HTTPCommunicationQueue.js
index 29c5f4b..1b71b04 100644
--- a/packages/@uppy/aws-s3/lib/HTTPCommunicationQueue.js
+++ b/packages/@uppy/aws-s3/lib/HTTPCommunicationQueue.js
@@ -295,7 +295,7 @@ export class HTTPCommunicationQueue {
};
for (;;) {
throwIfAborted(signal);
- const chunkData = chunk.getData();
+ const chunkData = await chunk.getData();
const {
onProgress,
onComplete,
@@ -401,7 +401,7 @@ async function _nonMultipartUpload2(file, chunk, signal) {
},
).abortOn(signal);
let body;
- const data = chunk.getData();
+ const data = await chunk.getData();
if (method.toUpperCase() === "POST") {
const formData = new FormData();
Object.entries(fields).forEach(_ref2 => {
diff --git a/packages/@uppy/aws-s3/lib/MultipartUploader.js b/packages/@uppy/aws-s3/lib/MultipartUploader.js
index e104fdc..8d79dc3 100644
--- a/packages/@uppy/aws-s3/lib/MultipartUploader.js
+++ b/packages/@uppy/aws-s3/lib/MultipartUploader.js
@@ -208,9 +208,13 @@ function _initChunks2() {
_classPrivateFieldLooseBase(this, _chunks)[_chunks] = Array(arraySize);
for (let offset = 0, j = 0; offset < fileSize; offset += chunkSize, j++) {
const end = Math.min(fileSize, offset + chunkSize);
- const getData = () => {
+ const getData = async () => {
const i2 = offset;
- return _classPrivateFieldLooseBase(this, _data)[_data].slice(i2, end);
+ return this.options.getChunkData
+ ? this.options.getChunkData(i2, end, {
+ chunkSize,
+ })
+ : _classPrivateFieldLooseBase(this, _data)[_data].slice(i2, end);
};
_classPrivateFieldLooseBase(this, _chunks)[_chunks][j] = {
getData,
@@ -228,7 +232,7 @@ function _initChunks2() {
}
} else {
_classPrivateFieldLooseBase(this, _chunks)[_chunks] = [{
- getData: () => _classPrivateFieldLooseBase(this, _data)[_data],
+ getData: async () => _classPrivateFieldLooseBase(this, _data)[_data],
onProgress: _classPrivateFieldLooseBase(this, _onPartProgress)[_onPartProgress](0),
onComplete: _classPrivateFieldLooseBase(this, _onPartComplete)[_onPartComplete](0),
shouldUseMultipart,
diff --git a/packages/@uppy/aws-s3/lib/index.js b/packages/@uppy/aws-s3/lib/index.js
index 88fd867..b6831cb 100644
--- a/packages/@uppy/aws-s3/lib/index.js
+++ b/packages/@uppy/aws-s3/lib/index.js
@@ -649,6 +649,7 @@ function _uploadLocalFile2(file) {
return _this.uppy.log(...arguments);
},
getChunkSize: this.opts.getChunkSize ? this.opts.getChunkSize.bind(this) : undefined,
+ getChunkData: this.opts.getChunkData ? this.opts.getChunkData.bind(this) : undefined,
onProgress,
onError,
onSuccess, |
|
Hi, can you explain more what the problem is you were facing and why we need a new option for it? |
|
@Murderlon hi , the idea itself is to control memory leak when uploading large file ie. 2 GB using multipart upload and avoid using large blob, the maximum created blob in memory has to equal chunk size so the example of realization will look // add file to uppy
`uppy.add({ size: 2Gb, originalPath: 'some_path' })`
// use aws s3 callback
uppy.use(Aws, {
....
getChunkData: ({ start, end }) => {
const b64 = RNFS.read(path, start, size 'base64')
// or use native java method to convert path to Blob using IOstream // for Android device similar for IOS
return new Blob([b64], { type });
}
}) |
|
What do you mean with "control memory leak"? Why isn't |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
preparing the solution to fix the data leak error for react native